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BACkGRjDUND AND PHILOSOPHY 

the CSE expository and narrative rating scales have been developed to 
meet the need for sound, ihstructidnally relevant methods tor assessing 
students' writing competence. Each scale's analytic rating elements are 
referenced to specific rhetorical features characteristic of the writing 
aim addressed exposition or harratidh. this referencing is based dh dur 
belief that students' writing in each mode can be analyzed into its consti- 
tuent elements, knowledge of students' performance with respect to these 
elements can provide valuable Infdnriaticn for assessing achievement and 
facilitating instructional ptaruning at the classroom, school, and district 
levels. 

the CSE analytic sciles reflect dur view df the roles of evaluation 
arid testing iri instructibri. We are convinced that testing and instruction 
can become complementary activities and that both need to share the same 
educatidnal goals and objectives. In the case of writirig, for example, 
criteria used to assess students' compositions ought to reflect agreed-upon 
standards of good writing. Classroom instruction should address explicitly 
these same staridards. Iri this kirid of system, assessment can play a valu- 
able role in instructional improvement by identifying specific areas of 
strength arid weakness for individual studerits, arid by iridicatirig areas 
where classroom, school, and district curricula maty need to be improved. 

the CSE scales employ analytic rating procedures to meet the demands 
df the kind df assessment system described above. Raters assigri poirits to 
each of several aspects of a student's composition, providing a rating of 
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the overall quality of the Written product as well as ratings bh specific 
elements. In contrast to the undifferentiated score provided by holistic 
scoring procedures, the CSE scales' overall rating and ratings on the 
specific elements give concrete information to guide instructional 
planning. 

fb be sure, some costs in rater time are involved in scoring writt€?h 
work analytically. Our studies indicate that it takes raters four to five 
minutes to assign analyt-^c ratings to a multi -paragraph writing sample ^ and 
two to four minutes for a single paragraph. In comparison, it may take 
only about one minute to assign a holistic rating to a fairly short writing 
sample. 

WhiVfe we believe that the greater time conmiitmeht required by analytic 
rating is offset by instructiorial advantages, economics may require some 
compromises. For example,, where student mastery and money are serious 
issues, the scale can be used to provide analytic ratings only for those 
students falling below mastery. In a similar vein^ the scale can be abbre- 
viated to target on specific skills of relevance to classroom instruction 
and ongoing classroom assessment. 

In the remainder of this paper^ we Will provide an overview to the 
scales and describe the theory and research fbrmihg their base. We will 
describe the features* purposes* and uses of the expository aiid the narra- 
tive scales, and the training that GSE provides for school systems wishing 
to use the scales in their writing assessment programs. 
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GVERVIEW TO THE SCALES 

We mentioned earlier that the GSE scales reflect the belief that 
students" writing can be analyzed into its constituent elements. We 
pointed out that if we can get accurate informatidn about students' perfor- 
mance oh these writing elements we can use it for instructional planning 
and Improvement. 

In the following sections we describe the theoretical and research 
basis for our belief that students* writing can be structurally analyzed. 
The work we describe l?d to the development of the analytic elements In our 
scales. 

TfielfieoceticaJ-^md Research^ Base 

Writing is a very complex skill. While most people would probably 
agree with that statement, some people may assume that "a good writer is a 
good writer" and that a student who writes well in one mode will write 
equally well in another. The research points out sonie dangers in that 
assumption. 

We can ask students to write in several modes of discourse or dis- 
course aims. The two modes most commbhly found In school curricula are 
expository and narrative writing. Expository writing involves expressing 
facts and ideas, and usually requires the student to support the ideas 
presented with appropriate detail, explanation, and J^^ical development of 
thought. Narrative writing relates experiences and events, and often 
requires the student to provide chronological development and description 
of events. 
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He believe that a student called on to write in the expository mode 
will need to draw on different skills^ reflecting different discourse 
structures, than he or she would need to achieve a narrative aim. For 
example, if a student is to explain something in the expository mode^ he or 
she would need to present the main idea, develop the idea with supporting 
details, and perhaps summarize the issue presented. On the other hand^ if 
a student is asked to give an account of something in the narrative mode?, 
he or she might heed to develop a sequence of events and describe the 
setting and characteristics with sufficient detail to make the account 
believable. 

The research findings support our view of these differing skill 
needs. There is a good deal of evidence from studies of students' reMia^ 
ability that different skitls are needed to understand text written in 
different modes. There is similar evidence from studies of students* 
writing ability that different skills are needed to pri3dij££ written text in 
different modes. 

For instance, a student reading a passage written in the expository 
mode relies oh a different set of organizing schemes, arid d-^ffererit methods 
of breaking dbwri, classifyirig, arid uriderstanding what is written than he or 
she would draw on to read a passage in the narrative mode (Meyer, 1975; 
Graesser^ Hauft-S^ith, Goheri, & Pyles, 1979). 

dust as students use differerit processes to read and make sen'ie of 
different kinds of writing^ they also use different skills when they are 
asked to write ih different modes such as exposition and narratiori (Veal i 
Tillmari; 1971; Praeter S Padia, 1980; Quellmalz, Capell , & Chou, 1982.) 
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We mentioned earlier that we are Interested In tying together testing 
arid Instruction. For other people sharing this Interest the research 
offers a clear message: Different kinds of writing asslgrimerits require 
different kinds of student writing strategies and skills. 

But there is a problem here. In many school s ' writing programs , 
students are normally asked to compose in only one mode ~ expository or 
rtarrative. Now^ a student might have greater ability in writing a narra- 
tive assignmerit stressing chrohblbgii:al development than he or she has in 
writing an expository assignment requiring l ogica l development. The oppo- 
site could also be true. 

A teacher might get one kind of picture, therefore, about a student's 
writing ability based on how he or she develops narrative. The teacher 
might get a different picture about a student's writing ability based oh 
how he or she develops an expository piece of work. If only one kind of 
writing ability is sampled, then it is possible that some writing defi- 
ciencies will go undiagnosed. 

The point we are making here Is simple: We cannot accept the assump- 
tion that a "good writer is a good writer." It depends on what we ask the 
student to write and for this reason we developed scales for the two dis- 
cburse aims most frequently used — expository and narrative (Quellmalz, 
1980). 

The eSE scales are impbrtarit, therefbre, because they provide ihfonna- 
tibh on students' expository and narrative abilities. They are equally 
important because of the way they assess these abilities. 

There are twb primary ways of assessing students' writing performahce 
— cbristhicted response arid selected response. A constructed response 
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provides a direct measure of a student's writing ability. Here* the 
student is asked to write something, perhaps a paragraph or two, perhaps an 
essay. When the piece of writing is turned in, a teacher or some other 
person judges the quality of the student's writing. 

A selected response provides ah indirect measure of a student's 
writing ability. Frequently, the student is given a passage to read 
followed by a multiple-choice test about the passage. This test may ask 
questions about the orgahizatibh of the written passage, its supporting 
evidence, its grammar and mechanics. The score the student gets on the 
test is supposed to show how much the student knows about writing. And so 
far* maybe i so good. But there is a problem if we make the inference that 
student knowledge of writing quality accurately indicates student ability 
to produce good writing. If this inference is unjustified, then once again 
students' writing deficiencies may remain andiagribsed. 

Several reasons are offered for using indirect^ multiple-choice tests 
of students' writing ability. First, multiple-choice tests take less time 
to score than essays do. Second, some people believe that there is a 
strong relationship between students' multiple-choice test scores and 
their Written work. Third, multiple-choice tests are more objective than 
ratings of students' written work. The argument is that if two people 
score the same essay in some general, impressionistic way, there is a good 
chance they will arrive at different judpehts about the same piece of 
Writing. Let us take up these three arguments. 

Argument one: Scoring essays does take more time than scoring a mul- 
tiple-choice test. However, bur scales have been designed so that they do 
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not take up exorbitant amounts of rater scoring time. Deperidirig eh the 
length of the writing sample and rater familiarity with the scale, rating 
l:ime ranges from two to five minutes. We believe the diagnostic and 
prescriptive information returned justifies the time invested. 

Argument two: There is no guarantee that a student who scores well 
on a mul tinle-chdice test of writing Will be able to produce good writing. 
Researchers working specifically in writing do not believe that such 
indirect measures as multiple-choice tests provide an accurate indication 
of students' writing ability (Braddoek, et al., 1963; Cooper S Odell, 
1977; Quellmalz, ^appell, & Chou, 1982). If these tests are well 
constructed (and that is always a big if no matter what their purpose is) 
then they may be fair measures of reading ^oiDprehehsj dri , but riot of how 
well a student will ^bduce a piece of writing . 

Argument three: The CSE scales have built-in procedures to make sure 
that different judges of a student's writing use the same decision rules. 
There is less liklihood, therefore, that they will give greatly different 
scores or grades to the same piece of work. 

At GSE we bijilt oh sorne of the research mentioned above in our own 
work on the different writing and response modes (Quellmalz, Capell, 4 
Chou^ 1982). In our studies, we looked carefully at whether students' 
writing abilities in the two major modes — exposition and narration — 
were comparable. We also examined whether different response modes — 
written work versus a multiple-choice test — provided the same kind of 
information about student writing ability. 
Here is a quick sketch of what we found: 

Students' writing skills did differ in the two discourse modes. 
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^ While ratings of exposition were generally different than those 
of narration, the mechanics scores on the two modes were 
comparable. Perhaps this helps explain the false notion 
that "good writing Is good writing." 

^ Mtil tiple-ehdiee scores were poorer indicators of student 
writing ability than measures based oh actual student 
writing samples. 

After these studies were conducted and the information analyzed, the 
scales we developed for the study were retained and now take the form we 
describe below. 
The-E xposl tdr y Sc a l e 

This scale, developed as part of CSE*s research on writing assessment 
(Quellmalz, 1980), is used to assess hdW well a student can write in the 
expository mode. By exposition, we have in mind writing that intends to 
inform by presenting facts ^ giving di recti onSi recording events, inter- 
preting facts, developing opinions. How well the student handles these 
elements influences the effectiveness of his or her expository composition. 

The scale uses five elements to assess students' writing in the expo- 
sitory mode. These are: General Competence; Essay Organization and Coher- 
ence; Paragraph Coherence; Support; arid Grammar/Mechariics. The first 
subscale. General Competence, Is used for making a holistic evaluation of 
the essay's command of basic writing elements. The four other subscales 
focus on an jlyti£_ eval uatlons of the quality of Writirig. 

Using the General Competence subscale, raters read the composition to 
form a global judgement of how well the student arranged fundamental 
elements. In rating the compositiori for its general quality, raters keep 
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In mind the question of whether the work achieves an expository purpose for 
the intended audience. General Competence asks the rater to keep the 



question of fundamental arrangement in mind and give an overall rating 
showing how well the student handles the skills incorporated in the four 
remaining subscales. The rater can assign the composition a global score 
ranging from six to one. Scores of six, five, and four represent varying 
degrees of mastery; scores of three, two, and one represent varying d€?grees 
of nori-mastery. This six-point system, which is described in greater 



detail later in the paper^ applies to all the subscales. 

In the remaining subscales, the rater's attention is directed to 
specific skills and qualities. Each skill or quality is described in 
detail. 

The Essay Qrgariizati on arid eoher^^^ focuses on the flow of 

ideas throughout the entire compositi oh and between its paragraphs. 
Emphasis here is on vertical relationships throughout the essay* 
The rater is reading to see if the essay has a main idea, for 
example^ which makes a point about tha topic ^whether that point is 
at a greater level of generality than the other points in the paper, 
and if the points made relate to the essay's thesis. 



The Paragraph Organization subsca with horizontal 

relationship within paragraphs, with the logical arrangement of 
points and their subordination to the paragraph topic. 

The Support subscaTe focuses on the specificity, depth, and amount 
of elaboration used to develop the theme. 

The Srammar/Mechahics subscale focu^ errors in sentence struc- 

ture or fliechanics and how seriously they interfere with communica- 
tion. It pinpoints global errors making it difficult to understand 
the Writer's message,^ and more local errors which are not serious 
impediments to communication. 



The Narrative Scale 

This scale, which was also developed as part of our research on writ- 
ing (Quellmalz, 1980), is used for assessing how we11 a student writes in 
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the narrative mode. By liarrati bh, we have In niirid a cfeseriptioh of an 
event(s), based on ah account of such aspects as (1) how, when^ and where 
the event(s) took place; (2) the persons, places, or things involved in the 
eveht(s); and (3) the actions, thoughts, or feelings of the actors 
described. How well the student handles these features influences the 
effectiveness of his or her narrative essay. 

The scale uses four elements to assess narrative writing. These are: 
General Competence; Focus and Grgahizatibh; Support; and Srammar/ 
Mechanics. The first subscale. General Competence, is used to make an 
overall or holistic evaluation of the essay's cdmmarid of writing fundamen- 
tals. The three other subseales are used to make analytic evaluations of 
the specific component features of narrative writing. Againi these three 
subseales focus on specific elements Which are described in detail. 

General Competence is based oh a global judgement of the compo- 
sition. The central question Is the narrative effectiveness of the work 
and how well it arranges the features incorporated in the remaining sub- 
scales. A six-point scale, with the same values as those described for the 
expository scale^ is used. 

The fbcus/0rgah1_2atibri subs concerned with the composition's 

structure and flow of Ideas— within- and between- paragraphs. 

Support^ which lias a similar emphasis to that desjcrl bed for exposi- 
tion, focuses the rater's attehtibh oh the development of events, 
descrlptlbns, and characters through the use of wel 1 -integrated 
details. 

The Granimar/Mechanlcs subscale focuses on the kinds of global and 
local errors In sentence structure and mechanics described above. 
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The research basis : Each scale, and its specific analytic elements^ 
is firmly anchored in our own and others' research on writing anc its 
assessment. The global or holistic judgment offered in each scale assumes 
that the whole is greater than the sum of its parts, and asfes for a rating 
of the quality with which the writer engages the topic to achieve the 
intended purpose. The discrete^ analytic features in each scale ask for 
separate ratings of a fundamental set of elements. These elements are 
specified because they recur in the research as basic features of acceptable 
writing. they also represent priorities in many writing competehey 
programs. The Individual subscales therefore present specific criteria for 
judging the student's skill In using these elements. 

Setting the scope and focus of the rssignment : Each scale is intended 
to be used in ah assessment setting in which students are provided with 
clear writing prompts conl:aining explicit directions to help them plan and 
develop their assignment, these directions should include: the specific 
purpose or function of the writing; a specific audience to whom the student 
will write; a specific topic or subject to write about; and criteria that 
will be used to judge the essay. 

These dlrectlohs are Intended to provide students with a clear and 
common understanding cif the task expected of them; that is^ the rhetorical 
context. As such, they are the Initial step In ensuring that each student's 
writing is judged on the same task description. They are a counterpart to 
the cdtnmdh scale criteria Used by raters. 

The six^^nl rating s e a l e : Each numerical rating on the scale Is tied 
to specific criteria and examples of skill levels and deficiency levels. 
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these numerical ratings are meariirigftil and cdrisisteht iridieatbrs of 
students' writing performance because they provide raters with specific 
guidance to decide upon the numerical rating that the composition should 
receive with respect to a given scale. 

Dn a given subscale a student receives a score indicating his or her 
degree of mastery of the skills specified for that scale. In exposition, 
for example, for a student to receive a rating of six (highest mastery 
level) on overall organization and coherence, the rater is directed to look 
for certain features in the composition. Among these features are: 
limitation of the topic; logical essay plan; maintenance of essay plan. 

For the student to receive a score of four (lowest level of mastery) 
on this part of the expository scale, among the features the rater is 
directed to look for are: clear main idea, logical plan, and reasonable 
support. 

Each of the other subscales used for expository rating offers similar 
criterion levels and examples. 

The same level of specificity of guidance to the rater is offered in 
the narrative scale. For example, on the support subscale, for a student's 
composition to receive a rating of five {second-highest mastery level) his 
or her work must provide supporting details, such as examples and descrip- 
tions, to develop events or characters. A student whose use of detail is 
not Integrated would receive a score of two. 

Adaptability te local eurri£Mliifi!_ Mld^ : The subscales, with their 
accompanying operational criteria, should represent instructional priori- 
ties of the school system using the scales. In an Ideally Integrated 
assessment and Instruction system, criteria used to evaluate writing 
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gathered for formal assessment should be the same as those used to evaluate 
and provide feedback to students on their more routine classroom writing 
assignments. While individual classrooms might focus on additional ele- 
ments such as originality of content arid style or voice, criteria used in 
judging formal, functional writing should reflect desired core writing 
elements. The CSE scales, therefore, include these commonly valued basic 
writing elements. 

These core elements are intended to provide starting points for 
schools and teachers to consider as they plan their own writirig assessmerit 
needs. For example, elaborated mechanics elements of sentence construc- 
tion^ usage^ spelling, punctuation^ and capitalization can provide useful 
classroom-level diagnostic information, especially for younger or less pro- 
ficient students whose writing shows various mechanical weaknesses. On the 
other hand, for large-scale competency testing programs assessing older or 
more able students, a cbmpreherisi ve mechanics rating may be sufficient. 

As we will see in the next section, when CSE provides schools and 
districts With training in the use of the scales, part of that training 
involves making sure the criterion descriptions are agreed upon by the 
potential raters. 

Copies of the complete, expanded scales arid scoring criteria appear 
In Appendix A. This expanded version serves primarily as a "text" which 
raters read before tralrilrig to familiarize themselves with the scales' 
rationale, structure, criteria, and criterion examples. During training 
and actual rating, raters use the shorter, tabular version of the scale^ 
which appears at the erid of Appendix A. Copies of actual student writing 
samples scored using our scales appear in Appendix 6. 
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TRAINING AND PROCEDURES 
TO ENSURE RELIABILITY 



In the preceding secion we mentioned that students need clear direc- 
tions to make sure that they all bring the same understanding of the 
writing task to the job at hand. We also pointed out that raters need to 
follow the same criteria as they assess students' written work* 

This section talks more about the procedures which help ensure that 
raters do indeed apply the same criteria. It also offers some reasons for 
why raters, even when they do try to use the same criteria, can "drift" 
apart as they go about the job of scoring students* written work. The 
training we offer helps dvercome this problem. 
Overview td_ tM ProJ)lems Md lheli^^o^ 



Samples of students* written work can provide direct evidence of 
writing ability. Scoring these samples, however, can present a problem, 
even with explicit criteria to follow; ho matter how good the scoring 
system, assuring reliability or agreement among raters requires careful 
plahhing. First, raters Who come to a training session with differing 
views of evaluation must accept and tlieh learn how to apply the rating 
scales accurately and consistently. Second, raters who have achieved high 
levels of agreement during training tend to drift apart and show less 
agreement when they must score large numbers of papers. This drift may be 
caused by fatigue leading to careless application of criteria, or it may be 
the result of other influences, such as the range of quality in the papers 
being scored or the re-emergence of idiosyncratic rater values. 
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What this means, then, is that even when a careful rating system is 
used, the more papers a rater reads the more likely he or she will drift 
away from the intended criteria. The training we provide, which grew out 
of previous CSE research and technical assistance in writing assessment 
(Quellmalz, 1980), has built-in procedures to control this potential drift 
and to keep raters on track. 
Training Sequence 

The training sequence has three basic features. First, it emphasizes 
developing consensus in defining scale elements that are sensitive to local 
needs. For example, the expository scale may present separate subscales 
for focus and organization rather than a combined coherence scale. Also, a 
district can decide to add features of concern in more advanced writing, 
such as style or attention to audience. 

Second, specific criteria are provided for each element, and raters 
receive repeated practice and feedback in their application. Third, the 
training serves as precursor to actual ratings of students' Written tests; 
it is hot merely an academic exercise. Therefore, the rating procedures 
are careful ly structured to ensure that raters 1 earn to apply cri teria 
accurately and continue to do so as they go oh to rate actual tei;t essays 
ihdepehder £ly. A key feature of tlis research-based training sequence is 
the use of essays which have been prescored by a panel of experts and which 
are then used as examples during training, as qualifying papers at the end 
of training, and as Inserted "check" papers during Independent scoring to 
Identify drifting raters and to help them stay on track. 
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Training proceeds as follows: 

Setting up a Rater Training Session 



Step 1: eSE and the school or district asking for the 
training discuss overall purposes* Questions may 
include: Is training to be provided for one scale 
or both? Will the scales be used for routine 
classroom assignments, school /district competency 
testing, or both? 

Step 2: The district collects representative's writing 
samples. These may come from pilot tests of prompts 
in neighboring districts. the samples must be 
available Well in advance (three, weeks) of the 
scheduled training session* 

The samples should reflect the spectrum of student 
writing abilities and represent the grade levels 
tested and/dr in which the raters teach. 

The sample/papers are used for three purposes: 

1. as practice training papers (approximately 30) 



2. as pilbttest papers to be sure that raters 
qualify at the end of training before proceeding 
to independent rating (approximately 20 to 30) 

3. as check papers that will__be inserted among the 
essays that will actually be judged during 
independent rating (a set of three papers to be 
read after every one or two hours of scoring) 



Step 3: Using the CSE scales CSE and district staff prescdre 
the samples for use as training, qualifying, and 
check papers* "Solid" and marginal examples rated 
from 1 to 6 on each of the subscales are prepared. 
"Feedback sheets" presenting the scdres and reasdh 
for the score that cite features of the papers and 
use language from the scale are prepared for al 1 
check papers arid for about 10 of the training 
papers. 
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Packets of the students' essaj^s are prepared for 
each rater. The training packet contains copies of 
the expanded, prose version and the shorter, tabular 
versidh of the rating scales, and the training 
essays* To acquaint raters witft the check procedure 
they Will encounter later, written feedback sheets 
are prepared for about 10 of the training papers i 
trainers, however^ have notes, to explain all the 
prescored training papers. Feedback sheets will 
only be distributed after raters have scored a 
paper. 

The qualifying packet consists of 10 to 20 prescored 
essays that represent the f yl 1 range of student 
cbmpdsitiohs. 



The check papers may be put in separate packets for 
each rater or be available at a central location. 

T^lt^jald^rs are prepared containing 10 essays which 
w1]l be rated ihdeperideritly._ The IQ papers in each 
folder are randomly selected from the full set of 
student essays to be rated. Each folder is 
numbered. 

A rater assignment sheet is prepared 1-istihg the 
sequence of folders that each rater who has 
qualified will score. Each folder of essays will be 
scored by two raters. These raters are randoitily 
assigned so that no two raters are systematically 
paired. 

Sheets for recording the scores that raters give to 
training papers* qualifying papers, and actual test 
papers are prepared. 



Prdvidihj the Rater Tralnjng 

The trairiihg_ begins wit^^^ a brief descHptioh of 
the scale's structure and rationale. Discussion 
deals with the broad features of the scale to be 
used* its research basis, and the need to develop 
common Understanding of purpose arid to maintairi 
high levels of rater agreement. 

Copies of the expanded scale are passed out and 
read by the raters. 
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The trainer and the raters discuss each element in 
the scale being used. Discussion includes the 
relationship between the scale's general 
cbmpeteriee rating and ratings on the subsequent 
subscales. Each of the subscales is defined and 
examples of the writing elements it focuses on are 
provided. 



Discussion then focuses on the six-point rating 
system applied to each subscale. Particular 
emphasi s is pi aced on the operati ojial cri teri a 
provided for each possible rating. The point of 
this discussion is to ensure cbmnibri understanding 
of each subscale and what each possible rating 
means. Any language clarification necessary is 
entered directly on the scale, which is then used 
fdf the remainder of: the training. The trainer 
points but that levels of rater agreement which 
are lower than .80 (that is, where raters are in 
lower than 80% agreesrentj will suggest less than- 
uniform scale application. 



The trainer passes out the prepared tralhirig 
packages of student writing samples. Raters are 
directed to read and then rate the first three 
writing samples in their package. The major 
intention here is for raters to begin "practicing" 
the criteria bh a feW papers. 

Discussion begins after raters have scored the 
first three training papers. The trainer presents 
the scores that the experts gave to these papers 
and explains the reason for these scores by 
reading through the essay and noting the features 
that led to the score it receivec: on each 
subscale. 



?2 
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Raters rate another three papers, and the trainer 
displays the scores that raters gave each essay, 
using the following tabular format: 



Rating 



1 


CM 


3 


4 


5 


6 



















































The trainer then presents the expert scores for 
these papers, and asks one or two individuals who 
gave scores that are two or more points away from 
the expert score to expl ai n thei r reasdrii hg by 
idehtifyihcL features in the papers that exemplify 
a_ score eategbry described in the scale. The 
trainer then explains the basis for the expert 
scores. 

Depending on the level of agreement among raters 
after they have scored the first six papers, 
discussion niay follow a variety of topics: 

is there need for further^ refinement of the 

language of the subscales? 
® is everyone bringing a eomrndn framework to the 

use of the scales? do seme raters have routine 

classroom diagnosis in mind while others are 

applyi ng the seal es from the standpoi nt of 

mi niimjm competency? 
® do these differing frameworks suggest more 

lehlent/mo re stringent application of criteria? 

is that appropriate? 

Raters then continue to rate training papers in 
sets of five. After iach set has been rated, the 
trainer distributes feedback sheets for two or 
three of the^ papers Iri the^^s^ These papers are 
discussed only !f questions are raised. Group 
discussion focuses on the other papers in the set^ 
following the strategy described in Step 5, and 
emphasizes those subscales where egreement levels 
are less than .80, th^t is, where less than 80% of 
the raters are in agreement. 
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After 10 to iS papers have been scored^ disetissibh 
and focus may beebme more direetrve. For example, 
if only a few individuals are giving discrepant 
scores, the trainer may work with these 
individuals while they are scoring a set. 

Step 8: When 80% agreement levels are reached bh the third 
or so set of training papers, the trainer may 
decide that it is time to distribute the 
qualifying papers. This set contains 15 to 20 
papers which raters score without interruption or 
discusGibn. To qualify, raters must agree with 
each other within bne point on each subscale on at 
least 80% of the papers. 

If most of the g^roup qualifies and is ready to 
proceed^ raters then begin the independent rating 
bf the actual student tef;t papers. 

At the same time, additional training may cbhtinue 
on a subscale with less than 80% agreement. For 
example^ individuals whb dbnbt agree at the 80% 
level may. cohtinue tb practice and discuss papers 
i ndi v1 dual ly or i h smal 1 groups wi th the 
assistance of the trainer. 

Independent Rating of Essays 

Step 1: Raters refer tb the rater assignment sheet to get 
the folders bf_essays they will score. After 
raters have scored one or two folders, they r^te 
the set of three check papers and record the 
scores they gave them. Th^ then turn in their 
check scores, and read the feedback sheets for the 
three papers. The pdinthere is to determine if 
the pair of raters scoring the same set of test 
papers are (1) in agreement with each other and 
(2) in agreemeht with tfte f cores given by the 
"expert" rateru Since students will actually be 
graded in the process, it is critical that raters' 
scores be no more than one point off the expert 
rater's score. Where greater differences exist, 
trainer and raters discuss the paper in question 
to resolve discrepancies. 
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Step 2: Depending bh the humber of actual student ^ssays 
to be rated, independent rating may continue over 
a few days, perhaps for a few hours each day. In 
this case, at the beginning of each rating day^^ 
raters may begin by reading and rating a set of 
check papers. This serves to keep raters bh track 
and to miriinrize the re-emergehce of idiosyncratic 
criteria. 

Summary 

Routine assessment of student's classroom progress may not require the 
careful check process described above. But it is another matter when large 
numbers of raters are to judge the writing performance of large numbers of 
students, say in a minimum competency testing program. Here the cheek 
procedure is critical, given the possibility of rater disagreement emerging 
over time. 

We strongly recommend, further, that when writing assessment results 
are to be used to make important decisions about student ability, each 
student's composition be read and scored, as described above, by two 
raters. If the two raters are in agreement at the .80 level or higher, the 
student's composition can be assigned the average of the two scores. If 
the two raters show agreement lower than .80, then the student's 
composition should be rated again by a third, independent judge. The two 
raters and the judge can then discuss differences via the kind of process 
described in Step 5 — citing features of the subscale element in the 
student's composition that match or exemplify the scale criteria for the 
score given. This discussion should lead to resolution of the rating 
discrepancy. 

Er|c ^5 
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CDNCLUSION 

In this paper we describe the need for using accurate and reliable 
criteria for the assessment of student writing. We offer scales for expo- 
sitory and narrative writing that we believe meet the standards of accurate 
and fair assessment. 

Both our research in the assessment of writing and the technical 
assistance we have provided in several settings indicate that large numbers 
of raters can be trained in the use of the scales, and that during training 
arid independent rating they can achieve arid maintairi high levels of 
agreement. Depending on the numbers of essays to be read and the numbers 
of people Who will be working as raters, the kinds of cheek procedures we 
describe are critical to the achievemerit and maintenarice of high levels of 
inter-rater reliability. 

It may be, at least initially and until raters become familiar with 
the scale procedures, that significant investments of time will need to be 
made. However, in the long run, if we are concerned about linking assess- 
ment with instruction so as to provide information for instructional 
improvement, the time invested can lead to high returns. 

For example, in schools where the CSE scales are used in the writing 
assessment program^ teachers state that the training and the scales have 
helped them to plari arid monitor instruction. They have become more aware 
of specific writing skills and are directing their instruction to scale 
elements that result in improved student writing. 
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Expository Scale 



Global Rating Procedures 

The purpose of global rating to form a single judgement af how well a 
piece of Jit ijbih communicates a whole message to the reader . Gl obal 
scoring assumes tni^E each characteristic that makes up an" essay — 
prganizatipn of ideas, content, mechanics, and so on --Is related to all 
other characteristics. It further assumes that some qualities of ah essay 
cannot easily be separated from each other. In short, the procedure views 
a piece of writing as a total work, the whole of which is greater than the 
sum of its parts. 

Discerning readers naturally will attend to^ or be Influenced by, some 
essay characteristics more than others. In this general scoring, however, 
readers should arrive at a Judperit regarding the essay's overall quality. 

For this element, you are being asked to form an overall judgement 
concerni ng the ef f ecti veness of the essays as exampl es of expdsi tory 
writing. The judgement should consider all the elements oh the scale ~ 
essay organization, paragraph organization, support, and mechanics. 

Some views oh exposition are given below: 

Exposition is the kind of discourse that explains or clarifies 
a subject. 



Exposition seeks to explain or inform through such methods as 
giving _ reasons or examples, comparing and contrasting^ 
defining, enumerating, or through a combination of methods. 

Exposition explains why or how. 

Exposi t1 bh promotes reader understanding of a subject. 



EEEHENT L 



Gene ral _ E omp ete nee 

Read each essay as a whole, first, in order to form an overall judgement of 
its quality. To assign the essay a score, consider the following 
qliestibhs: To what extent does the essay achieve an expository purpose for 
theihtehded audience? To what extent does the essay organize its elements 
to create an effective whole? 
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Assign each paper a mark of 1-6 using the scale below: 

MASTER 



6 = Very competent 



5 = Definitely competent 



4 = Adequately competent 



The paper executes all the elements 

competently. There are no serious errors. 

The paper has a clear main idea^ logical 

drganizatibn^ relevant, detailed su^^pbrt, 

and a cornmand of basic mechanics. There are 
ho major flaws. 

The paper is competent in all of the basic 

elements^ but there may be a few mihdr flaws. 

The paper is adequately competent in all 
elements. There may be a few flaws. Some 
may be serious.* 



NON-MASTER 



3 - Almost competent 



2 = Not very competent 



1 = Not at all competent 



The paper lacks competence in one jpir two 
elements, and there are several major flaws. 

The paper lacks competence in two or more of 
the elements. There are many serious flaws. 

Paper has none or only one of the elements 
competently executed. 



If the essay js only one paragraph, paragriiph cohesion is not considered 
a missing element, if the one existing paragraph coheres. If it clearly 
should have been divided into several paragraphs, then paragraph cohesion 
is a missing element. 
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ELEMENT 2 



EssayBrgani zatl^ii_aM_CoJiergiij:e 

This subscale fqc^^^ on the flow of ideas throughbut the entire paper and 
between paragraphs. The emphasis is on vertical relationships of ideas 
throughout the essay. 



The paper has a main idea (stated or clearly implied) which makes a point 
aboat the subject and is at a greater level of generality than the other 
points within the paper. Subtopics are logically related to the main idea 
and to each other. 

MASTER 

6 = ^ The subject is identified. _ 

^ The main idea is stated or implied in opening and/or closing 
statement. 

Opening and closing statements match or logically relate to the text 
and to each other. 

The topic is limited through reference to key points or lines of 
reasbningi _ ^ ^ 
The essay plan is logical . 

The essay plan is consistently maintained (no digression or 
extraneous materi al ) . 

5 = ^ The subject is ide'^.t^fiedi 

The main idea is stated or implied in bpening and/or closing 
statement. 

Opening and closing statements relate to or follow from the text and 
from each other. 

^ Thetbpic is partly limited by indicating number and type bf key 

points. 
^ Plan is logical. 

^ Plan is signalled by appropriate transitions. 
There niay be digressi bh or an elaboration. 

4 = ^ I7ie subject is identified. 

The main idea is identified or implied. 

There may or may not be art attempt to limit the topic, give 

directions to subsequent reasoning. 

There may be a few minbr digressions from the plan, but no major 
digressions. 

Subtopics can be reshuffled. 
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NGN-MASTER 

Subject is clear. 

Main point riiay hot, be very clear . There may be a major digression 

or several minor digressions. 

A plan is attempted which may need to be inferred. 

Subject is clear. 

Main idea is not very clear ahd/dr there may be more than one. 
The plan is attempted, but not consistently or not completely 
carried outi 

There are many digressions. 

Subject is unclear. 

Main idea is absent or very unclear. 

No plan is attempted or followed. 



32 
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ELEMENT 3 

Orjaanlzatldn - Paragraph* 

this subscale focuses on the relationship of ideas within paragraphs; their 
logical interrelationship and subordination to the paragraph topiCi 

Paragraphs present subtopics which are developed by cohesive groups of 
supporting statements. Each subtdpic represents a complete unit of 
thought. Major units of thought are delineated by physical separation of 
paragraphs. Statements within the paragraph relate logically to each other 
and to the paragraph subtopic. 

MASTER 

6 = ^ All major units of thought are set off by distinct paragraphs. 
TB¥ paragraph has a clearly stated or implied topic. 
All sentences within paragraphs are related to each other and to 
the paragraph topics arid are subordinate to it. There are no 
digressions or irrelevancies. There are no one-sentence paragraphs 
unless they are especially effective. 



5 = Most major subtopics are_ developed in paragraphs. 

^ ^ilT paragraphs cdhtaih idgically related subordinate support. 
® There may be a iiiirior digressiori. 

4 = ^ Many subtopics are developed in discrete paragraphs with related 
subordinate support. 
^ There mj be some minor digressidhs. 



MdN-MASTER 

3 = Ih some paragraphs statem^^ related, but may riot 

function as support subordinate to the paragraph topic. Paragraph 
separation is evident and consistent. Some relationships between 
sentences must be inferredc 

2 = ^ There are feW paragraphs where statemerits are logically related or 
supported. There are many digressions. Paragraph separation is 
inconsistent. Many relationships among sentences must be inferred. 

1 * ^ There are no paragraphs where statements logically cohere. 
Paragraph separation is incorrect. 



* For converitidris for paragraph separation (e.g.^ physical separation or 
indenting) see Mechanics. 
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ELEMENT 4 



Support 



This subscale focuses on the quality (specificity and relationship) of the 
support provided vertically for the essay theme as well as horizontally 
within each paragraph. 



)rt statements are at a greater level of specjficity than the generali- 
zations they are intended to support. Support statements logically relate 
to each other and to the generalization. Support includes specific details 
such as examples, facts, anecdotes, reasons, and concrete language. 



MASTER 

6 = ® The essay's main idea and all paragraph topics are supported by 
relevant^ specific statements. 

5 = The _essay's main idea an^ paragraph generalizations/ 

assertions are supported by predominantly specific statements. 
Enumerations are supported by descriptive detail, functions, or 
rationale. 

4 = ® The essay's main idea and most paragraph generalizations are 
supported. Most support is specific. Enumerations are supported by 
descriptive detail, functions^ or reasons. 



^ o 



NON -MASTER 

Some or all general izatiohs are supported by logically related 
detail , or some support is not speeifie but it is distinct and 
clear^ Support may be primarily an unelaborated, undetailed, 
unsupported list. 



2= An attempt is made to support general Izatioris/assertiohs. 

supporting sentences do not logically follow frorp each other or are 
reduhdanti 

)rt lacks precision, clarity In details, and/or language. 



1 = ^ No support is provided^, or^ 

® Supports if presents is vague i and confusing, or^ 
Not logically related to generalizations, or, _ 
At the same level of generality as the topic it attempts to support. 



ERIC 
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ELEMEMT 5 

Gramniar/Mgchsnlcs (Usage» sentence constructlorii spelling, punctuatlorii 
capitalization) 

Errors jri jgrammar or mechanics are considered according to how seriously 
they Interfere with cormnuni cation. These errors may be global or local . A 
global error makes it difficult to understarid the writerjs message. The 
sentence^ "tomorrow, I went to the store," for examp^le, forces the reader 
to decipher which time context (future or present) the Writer Is actually 
alluding to. A local error does not seriously interfere with the writer's 
message. For example, in the sentence, "He going to the store now," the 
message is clear but the grammar js Incorrect. Naturallyi^ an overabundance 
of errors which^^ if IMividually considered local, can seriously 

distract the r-^ader's attention and uhderstandlrig. The Intent here Is to 
evaluate errors in relattbri to how Jtiuch they interfere with the wrlter^s 
effectiveness in communicating rather tnan to attempt to assign different 
values to the myriad of possible grammatical and mechanical errors that can 
occur. 

MASTER 

6 = There are few or rib errors;. There are rib serious errors. 

5 = There may be a few minor errors In the categories, but no more than 
one serious error. 

4 = ® There are some errors. A few may be serious. 



NBN -MASTER 

3 = There are riumerbus errors in the categories. There are some 

serious errors in several categories ^ Below mastery in sentence 
construction. 

2 = * There are many serious errors, causing some cbnfuslori. 

1 = Errors are so numerous and serious that they Interfere with 
communication. 
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Check those mechanical skills below master level. 

Usage. Does not display coimnand of standard vocabulary usage. 

- Sentence construction. Does not display command of basic sentence 
structure. 

Spelling. Misspells many common words (includes homonyms). 

Gepitalizatiohs and pahctuation. _ Does^ n^^^^^ standard conventions 
appropriately, e.g., periods, commas, capitals, apostrophes. 

Conventions of paragraph separation are incorrect or inconsistent 

(indenting, spacing, titles, numbers). 

1) Usage 
Serious errors: 

- Homonyms, e.g.^ it, it's; their, there; to, two, too 

- Incorrect use of common words 

- Incorrect pronoun reference 

Minor errors: 

- awkward or odd use of words, phrases, but meaning still clear 

- vague, abstract language 

2) Sentence cdhstructtofl 
Serious Errors: 

- Subject verb agreement 

- Run dh 

- Frapents 

3) Spelling 
Serious errors: 

- Cpnmon words misspelled; does not include homonyms. Any misspelled 
word only counts as one error, even if the misspelling repeats. 

Minor errors: 

- Unusual, less frequent words 
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4) Capital Izatldn/Punctua tidn 

- Initial capitals --common proper nouns 

- Periods at end of sentence, common abbreviations 

- Commas (in series ^ for opening phrases) 

- Contractions 



5) Pa ragraph Cbnyentioni 
Serious errors: 

- Title 

- Nupibor 

- Inconsistency of separation convention 

- Absence of any convention for sepirction 

Minor errors: 

- Use of spacing instead of indentation (as in business letters) 



EKLC 
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Narrative Scale 



Blbbal Rating^ PcbcednrBS 

the purpose of global rating is to fortti an overall audgement of Kow well a 
piece of writing cornmunicates. a message to the reader. Global scoring 
assumes that eacK element of an essay is integrally related to other 
eleinents_ in the essay and the effect is cumulative. In short, the 
procedure views a piece of writing as a total work, the whole being greater 
than the sum of Its parts. 

For thiS element of "General Gdmpetence," you are being asked to form an 
overall judgement cone a coirpbsition's effectiveness as an example of 

narrative writing^* All of the elements presented in the scale--focus/ 
organization, support, and grammar/mechanics— should be considered in 
rating for general competence as well as any other elements which you may 
feel are important, 

ELEMENT 1 

Geflgral Impression 

Read each essay as a whole, first, to form ah overall judgement bfits 
quality. To assign the essay a score, consider the following questions: 
To what extent is the essay_ an example of effective narration? To what 
extent does the essay organize its elements to create an effective whole? 

MASTER 

6 = Ah excellent example of narration. Each element is evident in use 
throughout the essay (e.g., topic is clearly identified, characters 
defined, situations fully developed). 

5 = A good example of narration, but the elemejhts are hot equally well 
developed throughout the essay/paragraph. 

4 = An adequate example of_ narration, the writer incdrpdrates the ele- 
ments of narration. The essay is simple, informing and clear, and 
presents hbthing_ more than essentials. There may be one or two iso- 
lated instances of global error and no more than three local errors or 
a total of five local errors. 



NON-MASTER 

3 - A marginal example^ The writer presents evidence of limited skill in 
using the narrative elements. The elements are all present but 
developed poorly. Extremes are noted: strong to weak subjec^. focus, 
proper to poor grammar and mechanics. 

2 = A |Qdr example. WH^^^^ use of elements is problematic, focus on 
topic steadily decreases, support statements, if present, are weak; 
grammatical and mechanical errors also present. 

the paper is either off topic or gerirei. _Presents _hd central subject, 
supports are irrelevant, or absent, and contains numerous grammatical 
and mechanical rrors. the essay cannot be comprehended. 
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ELEMENT 2 

Focus/drganlzatlon 

This sdbscale examines whet^^ clearly indicated and developed 

in an organized manner. The composition should exhibit a clear structure 
both within and between paragraphs. The topic should be limited and free 
of extraneous material. 

MASTER 

6 = ^ The subject of the composition is clearly stated or implied. 
The topic is clearly limited; there are no digressions or 
extraneous material. 

Events arid/or ideas are presented ia a logical manner; they are 
linked both within andbetween paragraphs. 
^ The subject is developed through description ofevents* setting, 
or through the thoughts, emotions of the characters Involved. 



5 = The subject of the edmpdsitidri is clearly stated or implied. 

There may be one or two brief digressions or elaborations, but the 
topic is clearly limited. 
" Events and/or ideas are presents a logical manner; they are 
smoothly 11 niced both within and between paragraphs. 
The subject is developed through description of events, setting, or 
through the thoughts ahd/or emotions of the characters involved. 



4 = ^ The subject of the cbmpbsitioh is clearly stated or implied. 

There may be one or two brief digressions or elaborations^ but the 
topic is clearly limited. 

Events and/or ideas are presented in a logical manner, but linking 
within and/or between paragraphs may be weak in one or two 
lnstanees*_ 

The subject is developed through description of events,^ setting, or 
through the thoughts and/or emotiohs of the characters involved. 
The writer gives evidence of knowing the concepts of focus and 
drgahizatidn. 



NON-MASTER 

3 = ^ The subject ofthe cdmpdsitido is stated or implied. 

The logic or progression of ideas/events within or between 
paragraphs is often unclear; linking is frequently weak. 
The topic development through events, setting, or character 
thoughts/feelings is uneven; there may be too much or too 
little elaboration of some aspects. 
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The_ subject of the composition is stated or implied, but the main 
point is not dear. 

There are many digressions or elaborations; the topic is hot 
limited. 

The logic or progression of ideas ahd/br events is so unclear that 
no story line is discernible. 

There is little deyeloptnent through events ^ setting, or characters' 
feelings or thoughts. 

The subject of the composition is unclear or absent. 
There is no obvious organizational plan. 
There is no development of events, setting, or characters. 
There is much extraneous material. 
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ELEMENT 3 



This subscale fpcuses^n the quality (specificity and relationship) of_ the 
support provided for the essay theme both within each paragraph as well as 
throughout the essay as a whole. 

Support statements shoo^ at a greater level of specificity aird depth 
than the generalizations the to develop. Events* descrip- 

tions, and characters should be developed through the use of specific^ 
well-integrated details such as examples^ faets^ anecdotes, or descrtp- 
tioriSi Tliese details shouTd provide the reader with an image of the 
appearance, feelings, tJtoughts, actions, or mood of the events taking place 
in the narrative. 

MASTER 

6 = Events, charaater^^^^ developed by specific 

and clear suppDrting details, such as examples, descriptions, 
anecdotes 8 facts, etc. 

Suppprting details provide an imageyfeeling of actidhs, appearance^ 
feelings, thoughts, and/or mood in the paragraph/essay. 
® Supporting details are more specific than the general ideas/events ; 
characters are described arid well iritegrated to the rest of the 
paragraph/essay. 



5 = Events, characters^ and/or descriptions are developed by specific 
and clear suppcrting detail s^ such as examples, descriptions, 
iriecdbtes, fact£> eti. „ _ „ _ 

The use of suppirtirig detail Is riot consistent throughout* For 
example, in one or two instances, the writer may not provide infor- 
mation about ippsj-^ance, feelings* thoughts^ actions^ or inodd. 
Al though iiost supporting statements or details ^p 
descriptioris an^ are more specific than tlie geriera? stateprits they 
describe, there may be one or two iristarices where the detail lacks 
depth. 

4 = Except for one or two iristances, evcnits, characters^ and aetioris are 
described through the use of actjectives, adverbs, prepositional 
phraseSf etc. 

Events, chafaet^rsj and/or actions are developed by suppdrtirig 
detail • 

The use of supporting detail niay be inconsi stent or rudimentary i 
One aspect ifr^ritioned in the essay (i .e^^,^ xharaeter, everit, or 
description! my _riot be sufficiently developed. Overall, however^ 
the writer gives evidence of using supporting detail to develop most 
aspects of th^ essay/paragraph. 
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NON-MASTER 

® The use of detail is very uneven. Several statements/descriptions 

are not developed through the use of detail . 
^ There is tod ffiuch superficial or irrelevant detail 

Some of the detail is hot well integrated within the 

paragraph/essay. 

® there Is too little detail. There are very few instances where 
supporting detail is used to develop events, characters, or 

descriptions. J _ 

Host detail si acR specificity and depth. 

The details are hot smoothly integrated in the composition. 

Supporting details are vague, confusing, or hot related to the 
events, characters, or descriptidris they are meant to describe. 
There is little or ho evidence of supporting details in the 
paragraph/essay. 



42 



- 39 - 



ELEMENT 4 

G r a mma r /Mec han ics 

Errors in jrammar or mechanics are considered accdrdin seriously 
tfiey ihterf ere with GdnmiurLieatioh. These errors may be global or local, ft 
global error J^^ it difficult to -under sta^^ the writer Vs messuage. The 
sentience i ^Tomorrow, J went to the store," for example, forces the reader 
to decipher which time context (fu^^^^^ or present) the writer is actually 
alluding to. A local error does not seriously tnterfere with the writer's 
message. For example^ in the sehtehce^ "He going to ^the store now," the 
inessage is clear but the grammar is ihcorrecti^ Naturally, ah dverabuhdahce 
of errqri which^ if iridividuany considered are local ,_ can seriously 
distract the reader's attention and understanding. The intent here is to 
evaluate errors in relation to how much they interfere with the writer's 
effectiveness in communicating rather than to attempt to assign different 
values for the n^yriad of possible grammatical and mechanical errors that 
can occur. 

MASTER 

6 = the writer appears to have control of the usage and mechanical 
aspects of this essay, ^ _ ________ 

There are haiigl bbiuL errors . Recall that global errors affect the 
essay/paragraph as a whole and interfere with clarity of 
communication. 

^ There are only one or two local errors (if any) in mechaiiics. For 
example i there may be a few spel 1 i hg errors of difficult words, 
e.g. antenna. 

5 = Usage and mechanics are not a problem in this paper. 

® There are only a few local errftrs in usage or mechanics ^ 
There may J^^one_tsQla^ but the general meaning is 

clear throughout the paragraph/essay. 

^ Usage and mechanics are not a prdb^ ih this paper. 

Errors do not interfere with_ the clarity of communication. For 
example, confusion of to, too, two; their, there; or other local 
errors^ 

There may be one^ or two isolated instances of global lerrdrs, and no 
more than three local errors ^ dr a total of fivi local errors. 



NON-MASTER 

3« Some errars, global and local, do interfere with the clarity of 
communication. For examplei there may be a long run-bh sentence, 
inappropriate frapehts^ or incorrect tense continuity (e.g.. 
Yesterday we are going to school.") 
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Errors jitay detrac^^ of communication, such as 

confusing antecedents, omission of key vfords, serious misspellings 
of common words (e.g.^ confurmable/comfonable, laike/like) . 
There are many, global and local errors th bughout the 
essay/paragraph. 

Errors make thi s paper very di f f icul t to read and understand. There 
is an overabuhdance of global errors and a significant amount of 
CwaiDunicatioh is Ibst^ 

Errors are not restricted to one type of problem, such as run-on 
sentences. 
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Mechanlcs/Grarnmar 
Examples of Errors 



!• Sentence Eehstrtietieh 



Global errors 

- tense continuity {eog., Yesterday I go to ...) 

- very long run-on sentences 

- frapents 



Local errors 

- stibject/verb agreement (He go to the store,) 

- shortruh-bhs 

- incorrect or lack of connectors between clauses 
II. Usage 

Global errors 



- incorrect use of xo5pon words 

- code switching (mixing languages) (Today I played with my sister y 
fuimos to the store*) 

Local errors 

- homonyms (e.g., it, it's; their, there; to^ two, too) 

- incorrect pronoun reference (That is she book.) 

- awkward or odd use of words, phrases, but meaning still clear 



III. Spelling 

Global errors 



- Eofflffioo Words frequents does not include homonyms. Any 
misspelled wr^^^ one error, even if the misspelling 
repeats (comfortable-coni^urbal ; should nqt-shurent; wrong-rong) 

- Words misspelled so seriously as to impede communication (e.g., mild 
ddwn/melted down) 

Local errors 

- unusual, less frequent words 
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IV. Punctuation/Eapi tal i zati^ 
Global errors 

- periods at tHe end of sentences 

- contractions 

- commas where understanding is impeded 
Local errors 

- initial caps --common proper nouns 

- commas (in series, for opening phrases) 

- periods for abbreviation 



EKLC 
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TABULAR EXPOSITORY SCALE 



General 

^Qp^g"ge . Fbcds/Organizatlon Support Gralar/fleclianics 



0 suMeet clear L 

: maJn; Idea clear : . ; 

- key points, reasoning at beginning or end 
_ . ' beginning and end relate 
6 Excellent p 3nlog?cal 

1 Pan sipied^byransitlohs 

° paragraphs set $ff all major ideas 
, --; , no one sentence paragraphs 
(all) • noJigressions ^ 


main idea and all major 
topics: elaborated by 
specific details 

° enieratjons supported 
by deta; ], runciiofis, 
rationale 


^ one: Of two minor 
errors 

0 ~ 

no gross errors 

I 


: \ SUbJeCtiSleaF:! 

5 Good ° main idea ciear 

topic partly limited by reference to 
nufflberjr_type of key reasons 

• logical plan- 

• some iransitions _ 

. "'?st major ideas in paragraphs 
inost) • minor. digression 


" almost-all major points 
elaborated by specific 
details 

' ist elaboration is 
specific 


^ a few miner errors 

^ niAV his nhdi ii^neb 

iiiaj[ uc uiic yrusii 

error 

° usage and mechanics 
still not a problem 


• subject clear 

• iin idea cte: : 
_ ^ ' toplc-fflw be limited 

4 Adequate plan lopcal, but sub-topie can be 
, reshuffled : 

. many major thoughts in paragraphs 
, ■ , feSJiiior digressions, no major digres- 
(many) slons 


I many major points supported 
icn^jjaboration is 


I a few cdioh errors 
ope: or two gross 
D us no more than 
one minor error 


: subject clear 
, , , main idea not ver^ clear 
3 Developing I plali atteited,jus} infer 

.- some logically developed paragraphs 
,: . sp major digressions or excessive 
(some) elaboration 


** some ijor points develop- 
_ ed 

' some elaboration is 
specific but is distinct 

„ and clear 
liiay be a list 


° soiiie errors interfprp 

- tlVIUV VI 1 vl V III Vvi 1 Cl V 

some gross and 
^ mlnbrjrrors : 
sentence construc- 
tion below mastery 


I iuMeet clear 

4 n ji . 0 ^ii^ "^^.^^''y clear or more than one 
2 Rudimentary' plan atteped 

. ^ ' Few paragraphs logically developed 
(few) ' liiany digressions 


" supportive detail attempt- 

_ ed : 

I may be redundant 
majfinot be precise or 
clear 


' many gross and 
^ minor errors 
sbiiie confusion 


1 Off topic ; ml 
, Off genre 'pi 
(almost • al 
none) 


bjeetjayibe unclear 
in idea unclear 
an unclear 

most ho logically developed paragraphs 


° little or no support or 
" siipport^ls confusing 
or at the same level of 
generality as the main 
assertion 


' difficult to read 
many gross, varied 

^ errors 
very confusing 
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APPENDIX B 

Student Composj tiohs Rated 
During a Training Session 
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^ra4JF^^tng- Se ss i on 




Rated 



The five student writing samples in this appendix were rated during a 
eSE rater training session for secondary school teachers (Quellmalz, 
1982). The samples are representative lOth-grade student exposition, arid 
Were among those used early in the training as a check on how uniformly and 
consistently raters were applying the criteria. 

The rated writing samples show the level of agreement between tWo 
rater trainees. They also show how closely each seemed to be following the 
criteria agreed upon^ rather than applying perhaps more idiosyncratic 
values. 

At the top of each student composition are two numbers. These numbers 
show the General Competence rating that the two raters independeritly 
assigned to the sample. These general impressi bh ratings indicate each 
rater's assessment of how well the student's writing met the specific 
skills required iri Focus/Brgariizatibri, Support, arid Grammar/Mechanics. 

Each sample composition is followed by a feedback sheet showing a CSE 
rater's judgment of its general cdmpetericy level, as well as bf how well it 
dealt with the skills addressed in the three remaining subscales. 

As is readily seen, there are high levels of agreement between the two 
raters, arid similarly aceepitable agreement between the raters arid the 
assessments provided on the feedback sheets. Had these agreement levels 
riot been achieved^ the feedback sheets Would have beeri used as a focus for 
discussirig the causes bf rater disagreement and for resolving differences 
before taking up the next set of compositions to be rated. 
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. , S^t Muinbfir , 
-Code NuiiiEier 



EtBIENT 


RATING * ^ 


General Competency 


The nain point Is clear and specifically supported. 
4 Some mechanical errors and awicward usage. 


\ 

Focus/Organization 


riper nas main idea summarized in the opening 
and closing paragraphs. Ho use of transitions. 

4 


Support 


• 

4 Although there is only one example, it is clear 
and specific. * 


Grammar/Mechanics 


4 There are a few minor errors. "If a person... they." 
■Bralcie." 



Mechanical Skills below mastery 
Sentence Cohstructidh 



ERIC 



Spelling • 

Punctuation and Capitalization 
Paragraph Cohventldns 



1 H ■ 

V .. .l_-.lOth Drdca Prompt : 2 

... _ . : — .yfet-is DL prcrnibeS-'^.- promifie^-i^ j^Of?^^ __ 

An Ptnmplo. i^S; TP vir^ i p>rrvr>»c,^. >fo rl^r/r> — — 

. _ -HnL ^Tgfelg. nPf n(^i(^ I nnPf \//-^ jr pnrg^jnf ^ 

: — jQe)6anag^QMC-53.ta+eD7^/t. io-ixJccaiL^/iod 

. -IVYOugb-^e^a^.... ^ prr^nt^ Njf->j ip- r>i ^r> 

- :{^k-)J^ \|t^J nnnHo. n ^-vrynyg^.j 

. — it i^.itnpor+ant to keep j t... XP.. I'vjc^ : 

Wona^3. §ixxjd mjJtr.bH' ^fdmjn..ISz : : 

AJGM -pr^iii^ Sc^^-m-H-iing ^\)cu-^ou lcl 

. rla.»J^r-.^D. bach- ^sjout'i'-u^d . If _n^ou__ 

dlorff :._-rtiinte '^ou ccr^t K€€p.a. prorni^ ^oui.--- 

:^rDu\d-nD^ broHatq_^fe_Q^_feHot^e: 

.... - for-Q..dQle . davjj u:)han.JTfi^^Pr.i6r:)d 

ft:i-H-Nj|_u^Q^ over, ite ^ 

ge^ ym: rn^teup From .hiLr".. pur^ifl.^ .J 

<x pack cP pife pell cut . M}ten .. Ptitl-vj ^ ^ 

ae)hfl.d .. SiilH uS-Tit tKis^^g -u36rg..^^lly : 





._tec-^t€fe J.bJlJ^ . 



. .... upon -l^Iq . .fe^ciuXEjN^c^^ — .. . 
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ERIC 



Set Nu^^r _tra1n|ng Check 6_ 
Cods Humber 



C7) 



FEEBSaCfe SHEET- 



ELEKENT 



General Competency 



Fbciis/Organi za ti on 



RATING 



Paper is generally well developed, but lawed by 
poor paragraphing and numerous mechanieal errors. 



Position is clear and logically developed, but 
paragraphing is a problen. 
Good cbhclusidn. 



Support 



5 Gives a specific example to support thesis. 



Grammar/Mechani cs 



3 liuinerous awkward usages. 



Prdblem with use of cdmnas, 



Mechanical Skills below mastery 

Sentence eonstructioh 

y Usage 
Spelling 

. • — y Punctuation and Capitalization 
ParagrapH Conventions 



.^5 




_ " ' 1 _ _ _ : 






^^ ij> ^ . .c: fv/T^^ . j^"^ ;.C . ^\ . \ ^^rd^ \rr\ 







Set Number _ Training Check.? 
Codu Number - 



FEEDBACK 5HEET 



ELEflENT 


V 

RATING 


General Gompetency 


The line of reasbhirig is confusing. Support is 
vague and mechanics are intrusive. 


\ 

\ 

Fdcus/Orgahizatibh 


• 

The main Idea Is unclear and there is no conclusion. - 
2 Paper moves from "definition of promises" to 
"Very often they should be broken." 
No evidence of paragraphing. 


Support 

• 


# 

2 Support Is attempted, but not specific. 


Grammar/Mechanics 


Sentence errors. : ~ 
3 Some awkward usage. "We use promises..." 

"£ik6 when..." "Get caught in the middle of two 
people." 



.Mechanicat Skills w mastery 

^ Sentence Cbhstructlbn 



SErJc'32 



Spelling 

Punctuation and Capitalization 
ParagrapH Conventions 



Set Number - Irmn^ 

Code Number 



(5) 



EtEJIENT 


RATING 


General Competency 


Main point confusing because of poor logic and 
2 rit^erbus mechanical errors. 


\ 

Focus/Organi zati on 


2 the logic is unclear. The paper confuses promises With 
lying. 


Support 


2 Support is attempted, but unclear. 


Granimar/Mechani cs 


Major usage» sentence construction and spelling 
1 errors interfere with meaning. 



Mechanical Skills below mastery 

^ Sentence Construction 
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_ Spelling 

Punctuation and Capitalization 
Psragi iph Conventions 
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"uvnfe:. . s^-S._-_.s*Aii^«KJt^o ...... .CrtCE.. r 
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Code Number 



ELBIENT 


RATING 


General Competency 


3 The papers' point is at times confused by 
Imprecise language. 


Focus/Organization 


Logical development Oiiclear. 

"promise = blackmail" 

"promise = are big and small things" 


Support 


2 Support is not very specific or clear. 


Graircnar/Meehahies 


3 Usage is a pervasive problen. "For how oh"* "Like 
she wanted me to. . slang terms. 
Subject-verb agreement. 



Mechanical Skills below mastery 

, Sentence Construction 

^ Usage 
Spelling 

• Purietuation and Capitalization 
Paragraph Conventions 
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